Methods and apparatus for facilitating access to content in a data storage system

ABSTRACT

One embodiment is a system for locating content on a storage system, in which the storage system provides a location hint to the host of where the data is physically stored, which the host can resubmit with future access requests. In another embodiment, an index that maps content addresses to physical storage locations is cached on the storage system. In yet another embodiment, intrinsic locations are used to select a storage location for newly written data based on an address of the data. In a further embodiment, units of data that are stored at approximately the same time having location index entries that are proximate in the index.

BACKGROUND OF INVENTION

1. Field of Invention

The present invention relates to methods and apparatus for accessingcontent in a storage system.

2. Description of the Related Art

Content Addressed Storage (CAS) is a technique by which a unit of datastored on a storage system is accessed using an address that is derivedfrom the content of the unit of data. As an example, the unit of datamay be provided as an input to a hashing function which generates a hashvalue that is used as the content address for the unit of data. Anexample of a hashing function suitable for generating content addressesis the message digest 5 (MD5) hashing algorithm. When a host computersends a request to a content-addressable storage system to retrieve aunit of data, the host provides the content address (e.g., hash value)of the unit of data. The storage system then determines, based on thecontent address, the physical location of the unit of data in thestorage system, retrieves the unit of data from that location, andreturns the unit of data to the host computer.

The task of determining the physical location of the unit of data mayinvolve several aspects, particularly when the storage system is adistributed storage system. A distributed storage system is one made upof a number of separate nodes, where each node may be a separate machinewith separate resources (e.g., processor, memory, disk). The nodescommunicate with each other (e.g., through a network) to handle dataaccess requests from one or more host computers. To determine thephysical location of a unit of data on the storage system based on thecontent address of the unit of data, the storage system first determineson which node the unit of data is stored. Then, the storage systemdetermines which disk on that node the unit of data is stored (if thenode has multiple disks), as well as the physical location on the diskat which the unit of data is stored (e.g., cylinder, head, sector).

FIG. 1 shows an example of a distributed storage system. The distributedstorage system includes a plurality of access nodes (AN) 101 a, 101 b, .. . , 101 n, a plurality of storage nodes (SN) 105 a, 105 b, 105 c, . .. , 105 n, and a network 103 that couples them together. Access nodes101 may be used to process access requests (e.g., read/write requests)from host computers (not shown), while storage nodes 105 may be used tostore data. When an access node receives a request from a host to read aunit of data, the access node determines on which storage node(s) theunit of data is stored, and requests the unit of data from theappropriate storage node(s).

One known method of determining which storage node stores a particularunit of data is referred to herein as a multicast location query (MLQ).In a multicast location query, an access node 101 receives a request toaccess a unit of data from a host. The access node then broadcasts anetwork message to each storage node 105, asking if the storage nodestores the particular unit of data. Each storage node 105 thendetermines if the requested unit of data is stored thereon. Each storagenode 105 may include a data set (e.g., a database or table) that liststhe content addresses of the units of data stored on that storage node,along with the disk in the storage node on which the unit of data isstored. That is, if the storage node has four disks, the table mayindicate on which of the four disks each unit of data is stored. Thus,when a storage node 105 receives the MLQ network message from an accessnode 101, the storage node may scan its data set to determine if therequested unit of data is stored thereon.

Once a storage node 105 determines that it stores the requested unit ofdata, the storage node, using the same data set, may determine on whichphysical disk the unit of data is stored. Then, the storage node maydetermine the physical location on the appropriate disk at which theunit of data is stored. The units of data may be stored as files in afile system on the storage node. Thus, to determine the physical disklocation of a unit of data, the storage node may locate thecorresponding file in the file system that includes the data unit andrely on the storage node's operating system to map the file systemlocation to a physical disk location. For example, when storing a unitof data, the storage node may create a file having the content addressof the unit of data as its filename and store the unit of data in thatfile.

FIG. 2 is an illustrative file system for storing units of data in adistributed content-addressable storage system. The file system of FIG.2 includes a number of hierarchical directories. The directory at thetop of the hierarchy is termed the root directory. At the second levelin the hierarchy are a number of subdirectories. Each of thesesubdirectories represents the first character in the content address ofa unit of data. That is, a unit of data having a content addressbeginning with the character ‘A’ will be stored in one of thesubdirectories of directory ‘A.’ The subdirectory in which the unit ofdata will be stored is dependent on the second character of the contentaddress. When the storage system later attempts to access the unit ofdata (e.g., in response to a read request), the storage system maylocate the unit of data by traversing the file system hierarchy tolocate the subdirectory whose name matches the first two characters ofthe content address of the unit of data. If the storage system locatesthe unit of data in its file system, it may open the file containing theunit of data to verify that the storage node does indeed have the unitof data stored thereon. Then, the storage node may return the unit ofdata to the access node that issued the multicast request. The accessnode may then return the unit of data to the host that requested theunit of data.

An MLQ is a computationally expensive process, as it requires eachstorage node to perform an exhaustive database search for each unit ofdata requested. Further, most of these exhaustive searches will fail, asa unit of data typical resides on only one or a small number of storagenodes (assuming the unit of data is replicated on one or more storagenodes).

To reduce the computational expense of using an MLQ to locate units ofdata on the storage system, another technique has been developed thatemploys an index to locate units of data. The index is referred toherein as a blob location index (BLI), with the term “blob” referring toa unit of data The BLI is a database that maps the content addresses ofunits of data (“blobs”) to the storage node or nodes on which thecontent is stored. In much the same manner as in the MLQ scheme, unitsof data are stored in a location in the file system selected based onthe content address of the unit of data. However, the administration ofthe BLI is split among the storage nodes, so that each storage nodeadministers a portion of the BLI. Thus, access requests for a unit ofdata need not be broadcast to all storage nodes, but just to the onethat administers the portion of the BLI that includes the requested unitof data.

A configuration of the BLI is shown in FIG. 3. The responsibility ofadministering the BLI is split evenly across storage nodes 301, 303,305, and 307. Storage node 301 administers the portion of the BLI thatcontains content addresses beginning with characters ‘A’-‘F’, storagenode 303 administers the portion of the BLI that contains contentaddresses beginning with characters ‘G’-‘L’, storage node 305administers the portion of the BLI that contains content addressesbeginning with characters ‘M’-‘R’, and storage node 307 administers theportion of the BLI that contains content addresses beginning withcharacters ‘S’-‘Z’. Each portion of the BLI includes an entry for everycontent address within the specified range that is stored on the storagesystem, and indicates on which storage node the corresponding unit ofdata is stored. The storage nodes also have local databases 309, 311,313, and 315, which store the content addresses of units of data storedon their respective storage nodes and indicate on which physical disk ofthat storage node these content addresses are stored.

The access nodes maintain a record of which portions of the BLI areadministered by each storage node. Thus, when an access node receives arequest from a host to retrieve a particular unit of data, the accessnode determines which storage node administers the portion of the BLIthat contains the content address of the requested unit of data. Forexample, if a host sends a request to an access node for a unit of datahaving a content address beginning with ‘S’, the access node queriesstorage node 307 to determine which storage node stores the requestedunit of data. Storage node 307 searches the BLI to determine whichstorage node or nodes store the requested unit of data and returns thisinformation to the requesting access node. The access node may requestthe unit of data directly from the appropriate storage node. In thismanner, other storage nodes that do not store the unit of data are notqueried. Thus, unlike an MLQ, using the BLI does not require eachstorage node to perform an exhaustive database search. Instead, only onestorage node queries the BLI, and one storage node queries its localdatabase, thereby reducing the overall computational expense on thestorage system.

In case of a failure in the BLI to return the storage node for arequested unit of data (i.e., if a content address requested by a hostis not found in the BLI), the storage system may fall back on the MLQscheme, and issue an MLQ to determine on which storage node the unit ofdata corresponding to the requested content address resides.

By distributing the BLI administration responsibilities evenly among thestorage nodes, the computational burden of locating a particular unit ofdata on the storage system is shared equally among the storage nodes.The randomness of the hashing function used to generate the contentaddresses is relied upon to distribute an approximately equal number ofcontent addresses to each storage node. When new storage nodes are addedto the storage system or when storage nodes are removed from the storagesystem, the administration responsibilities of the BLI are redistributedamong the storage nodes so that the administration responsibilities areevenly shared among all the storage nodes. Further, when new units ofdata are stored on the storage system, the storage node updates the BLI,updates the local database of the storage node on which the unit of datais stored, and writes the content itself to the storage system. Thisthree-tiered write impacts the performance of the storage system inprocessing writes.

SUMMARY OF INVENTION

One illustrative embodiment is directed to a method of processing datain a computer system comprising at least one host and at least onestorage system which stores data for the at least one host. The methodcomprises acts of: (a) receiving, at the storage system, a request fromthe at least one host to access a unit of data stored on the at leastone storage system; and (b) in response to the request, returning to theat least one host information related to where the unit of data isphysically stored on the at least one storage system. Anotherillustrative embodiment is directed to at least one computer readablemedium encoded with instructions that, when executed on a computersystem, perform the above-described method.

A further illustrative embodiment is directed to a storage system foruse in a computer system that includes at least one host, wherein thestorage system stores data for the at least one host. The storage systemcomprises: at least one storage device to store data received from theat least one host; and at least one controller that: receives a requestfrom the at least one host to access a unit of data stored on thecontent addressable storage system, the request identifying the unit ofdata using a content address, and in response to the request, returns tothe at least one host information related to where the unit of data isphysically stored on the content addressable storage system.

Another illustrative embodiment of the invention is directed to a methodof processing data in a computer system comprising at least one host andat least one storage system which stores data for the at least one host.The method comprises acts of: (a) sending, to the at least one storagesystem, a request to access a unit of data stored on the storage system;and (b) receiving, from the at least one storage system, informationrelated to where the unit of data is physically stored on the at leastone storage system. Another illustrative embodiment is directed to atleast one computer readable medium encoded with instructions that, whenexecuted on a computer system, perform the above-described method.

A further illustrative embodiment is directed to a host computer for usein a computer system that includes the host computer and at least onestorage system, wherein the host computer accesses data stored on the atleast one storage system. The host computer comprises: at least onestorage device; and at least one controller that: sends, to the at leastone storage system, a request to access a unit of data stored on the atleast one storage system; and in response to the request, receives fromthe at least one storage system, information related to where the unitof data is physically stored on the at least one storage system andstores the information in the at least one storage device.

Another illustrative embodiment is directed to a method of processingdata in a computer system comprising at least one host and at least onestorage system. The method comprises acts of: (a) receiving, at the atleast one storage system, a request from the at least one host to accessa unit of data, the request having an identifier that can be used toaccess the unit of data; and (b) in response to the request, returningto the at least one host an identifier that can be used to access theunit of data and additional information, separate from the identifier,related to where the unit of data is physically stored on the at leastone storage system. Another illustrative embodiment is directed to atleast one computer readable medium encoded with instructions that, whenexecuted on a computer system, perform the above-described method.

A further illustrative embodiment is directed to a storage system foruse in a computer system, including the storage system and at least onehost. The storage system comprises: at least one storage device to storedata received from the at least one host; and at least one controllerthat: receives, at the at least one storage system, a request from theat least one host to access a unit of data, the request having anidentifier that can be used to access the unit of data; and in responseto the request, returns to the at least one host an identifier that canbe used to access the unit of data and additional information, separatefrom the identifier, related to where the unit of data is physicallystored on the at least one storage system.

Another illustrative embodiment is directed to a method of processingdata in a computer system comprising at least one host and at least onestorage system. The method comprises acts of: (a) sending, to the atleast one storage system, a request from the at least one host to accessa unit of data, the request having an identifier that can be used toaccess the unit of data; and (b) receiving, from the at least onestorage system, an identifier that can be used to access the unit ofdata and additional information, separate from the identifier, relatedto where the unit of data is physically stored on the at least onestorage system. Another illustrative embodiment is directed to at leastone computer readable medium encoded with instructions that, whenexecuted on a computer system, perform the above-described method.

A further illustrative embodiment is directed to a host computer for usein a computer system including the host computer and at least onestorage system. The host computer comprises: at least one storagedevice; and at least one controller that: sends, to the at least onestorage system, a request from the at least one host to access a unit ofdata, the request having an identifier that can be used to access theunit of data; and in response to the request, receives from the at leastone storage system an identifier that can be used to access the unit ofdata and additional information, separate from the identifier, relatedto where the unit of data is physically stored on the at least onestorage system, and stores the additional information on the at leastone storage device.

Another illustrative embodiment is directed to a method of processingdata in a computer system comprising at least one host and at least onecontent addressable storage system which stores data for the at leastone host, wherein the at least one host accesses data units stored onthe at least one storage system using content addresses generated basedon the content of the data units. The method comprises: (a) maintainingat least one index that maps a content address of at least one data unitto a storage location within the at least one storage system at whichthe data unit is stored; and (b) maintaining a cache of the locationindex. Another illustrative embodiment is directed to at least onecomputer readable medium encoded with instructions that, when executedon a computer system, perform the above-described method.

A further illustrative embodiment is directed to a content addressablestorage system for use in a computer system, that includes the contentaddressable storage system and at least one host, wherein the at leastone host accesses data units stored on the content addressable storagesystem using content addresses generated based on the content of thedata units. The content addressable storage system comprises: at leastone storage device to store data received from the at least one host;and at least one controller that: maintains at least one index that mapsa content address of at least one data unit to a storage location withinthe content addressable storage system at which the data unit is stored;and maintains a cache of the location index.

Another illustrative embodiment is directed to a method of processingdata in a computer system comprising at least one host and at least onecontent addressable storage system which stores data for the at leastone host, wherein the at least one host accesses data units stored onthe at least one storage system using content addresses generated basedon the content of the data units. The method comprises an act of: (a) inresponse to an access request from the at least one host computer for aunit of data identified by a content address, parsing the contentaddress to determine at least one aspect of a physical storage locationfor the unit of data on the at least one storage system. Anotherillustrative embodiment is directed to at least one computer readablemedium encoded with instructions that, when executed on a computersystem, perform the above-described method.

A further illustrative embodiment is directed to a content addressablestorage system for use in a computer system, including the contentaddressable storage system and at least one host, wherein the at leastone host accesses data units stored on the content addressable storagesystem using content addresses generated based on the content of thedata units. The content addressable storage system comprises: at leastone storage device to store data received from the at least one host;and at least one controller that, in response to an access request fromthe at least one host computer for a unit of data identified by acontent address, parses the content address to determine at least oneaspect of a physical storage location for the unit of data on the atleast one storage system.

Another illustrative embodiment is directed to a method of processingdata in a computer system comprising at least one host and at least onecontent addressable storage system which stores data for the at leastone host, wherein the at least one host accesses data units stored onthe at least one storage system using content addresses generated basedon the content of the data units. The method comprises acts of: (a)receiving, from the host, a request to store a unit of data on thestorage system, the unit of data having a content address based on thecontent of the unit of data; (b) determining, based on the contentaddress, a first storage location on the storage system to which thecontent address maps;

(c) storing a pointer for the first unit of data at the first storagelocation, the pointer pointing to a second storage location; and (d)storing the unit of data at the second storage location on the storagesystem. Another illustrative embodiment is directed to at least onecomputer readable medium encoded with instructions that, when executedon a computer system, perform the above-described method.

A further illustrative embodiment is directed to a content addressablestorage system for use in a computer system that includes at least onehost, wherein the at least one host accesses data units stored on thecontent addressable storage system using content addresses generatedbased on the content of the data units. The content addressable storagesystem comprises: at least one storage device to store data receivedfrom the at least one host; and at least one controller that: receives,from the host, a request to store a unit of data on the storage system,the unit of data having a content address based on the content of theunit of data; determines, based on the content address, a first storagelocation on the storage system to which the content address maps; storesa pointer for the first unit of data at the first storage location, thepointer pointing to a second storage location; and stores the unit ofdata at the second storage location on the storage system.

Another illustrative embodiment is directed to a method of processingdata in a computer system comprising at least one host and at least onecontent addressable storage system which stores units of data for the atleast one host, wherein the at least one host accesses a unit of datausing a content address based at least in part on the content of theunit of data. The method comprises an act of: (a) creating the contentaddresses for units of data to comprise first information that providesan indication of which units of data are written to the storage systemproximate in time. Another illustrative embodiment is directed to atleast one computer readable medium encoded with instructions that, whenexecuted on a computer system, perform the above-described method.

A further illustrative embodiment is directed to a host computer for usein a computer system comprising the host computer and a contentaddressable storage system, wherein the host computer accesses the dataon the content addressable storage system using content addressesgenerated based on the content of the data units. The host computercomprises: at least one processor that generates the units of data; andat least one controller that creates the content addresses for units ofdata to comprise first information that provides an indication of whichunits of data are written to the content addressable storage systemproximate in time.

Another illustrative embodiment is directed to a method of processingdata in a computer system comprising at least one host and at least onecontent addressable storage system which stores units of data for the atleast one host, wherein the at least one host accesses a unit of datausing a content address based at least in part on the content of theunit of data, wherein the at least one storage system comprises an indexthat maps content addresses for units of data to storage locationswithin the at least one storage system where the units of data arestored. The method comprises acts of: (a) receiving, from the at leastone host, a request to store one of the units of data, the requestidentifying the one of the units of data with its associated contentaddress; and (b) storing the one of the units of data in a storagelocation selected so that an entry in the index for the storage locationis proximate to entries in the index for other units of data written tothe at least one storage system proximate in time to the one of theunits of data. Another illustrative embodiment is directed to at leastone computer readable medium encoded with instructions that, whenexecuted on a computer system, perform the above-described method.

A further illustrative embodiment is directed to a content addressablestorage system for use in a computer system that comprises the contentaddressable storage system and at least one host, wherein the at leastone host accesses data units stored on the content addressable storagesystem using content addresses generated based on the content of thedata units. The content addressable storage system comprises: at leastone storage device to store data received from the at least one host;and at least one controller that maintains an index that maps contentaddresses for units of data to storage locations within the at least onestorage system where the units of data are stored so that units of datawritten to the at least one storage system proximate in time haveproximate entries in the index.

BRIEF DESCRIPTION OF DRAWINGS

In the drawings:

FIG. 1 is a block diagram of an illustrative prior art distributedstorage system;

FIG. 2 is a diagram of a file system hierarchy in which units of dataare stored in a prior art content-addressable storage system;

FIG. 3 is a block diagram illustrating a blob location index sharedamong multiple storage nodes in a prior art content-addressable storagesystem;

FIG. 4 is a block diagram illustrating a storage system and hostconfiguration on which aspects of the invention can be implemented;

FIG. 5 is a block diagram illustrating intrinsic locations assigned to aplurality of storage nodes in accordance with one embodiment of theinvention;

FIG. 6 is a block diagram illustrating intrinsic locations assigned to aplurality of storage devices within a storage node, in accordance withone embodiment of the invention; and

FIG. 7 illustrates locality information being added to the contentaddress for a unit of data in accordance with one embodiment of thepresent invention.

DETAILED DESCRIPTION

Aspects of the present invention relate to improved techniques foraccessing content in a storage system.

In accordance with one embodiment of the present invention, the storagesystem may provide hints to the host as to the location where a unit ofdata is stored within the storage system. On subsequent access requeststo the unit of data, the host can provide the location hint back to thestorage system, to facilitate the storage system locating the unit ofdata, and thereby increase the efficiency of the storage system inlocating the unit data.

In an alternate embodiment of the present invention, acontent-addressable storage system is provided with an index that maps acontent address of at least one unit of data to a storage location atwhich the unit of data is stored. The storage system maintains a cacheof the location index to achieve performance improvements in accessingthe information contained within the index.

In a further embodiment of the present invention, a technique isemployed for facilitating the processing of a request from a host toaccess a unit of data in a content-addressable storage system. Thetechnique includes parsing the content address to determine at least oneaspect of a physical storage location at which the unit of data isstored. By employing at least one aspect of the content address to bedeterminative of at least one aspect of the physical storage locationwherein the unit of data can be stored, locating a unit of data inresponse to an access request can be achieved efficiently, as discussedbelow.

In another embodiment of the present invention, a technique is employedto create content addresses for units of data, such that the contentaddresses include information that provides an indication of which unitsof data were written to the storage system proximate in time. As will beappreciated from the discussion below, providing content addresses thathave some similarity in at least a portion thereof for units of datathat are written proximate in time can increase efficiencies in astorage system locating units of data in response to access requestsfrom a host.

As should be appreciated from the foregoing, several of the aspects ofthe present invention are specifically directed to content-addressablestorage systems. However, this is not true for other aspects of thepresent invention, which have broader applications. Thus, although thespecific implementations described below relate particularly to acontent-addressable storage system, it should be appreciated that allaspects of the present invention are not limited in this respect, asthey can be employed on other types of storage systems.

An illustrative computer system configuration on which aspects of thepresent invention can be employed is shown in FIG. 4, which illustratesa host computer 401 coupled to a storage system 407.

Host computer 401 executes an application program 403 that reads andwrites units of data, and an application programming interface (API) 405that is adapted to interface with the storage system 405. Host 401 maybe, for example, a server that provides resources (e.g., data storage,email, and other services), a client (e.g., a general-purpose computersystem operated by a user), a network component (e.g., a switch) or anyother type of computer. Host 401 is coupled by a communication link 404to storage system 407 so that the host 401 can use the storage system407 to store and retrieve data. The link 404 can be any type ofcommunication link, including any type of network or direct connection.

Application 403 may be any computer application program that a user oradministrator of host 401 uses and that stores data to and/or retrievesdata from storage system 407. Application program 403 is linked with API405 so that application program 403 may use API 405 to communicate withstorage system 407.

API 405 may include, for example, computer object code that is providedto the programmer of application program 403. The computer object codeincludes routines that may be used to communicate with the storagesystem. When linked with API 405, an application program may call theseroutines to communicate with storage system 407. API 405 may be linkedwith any number of applications, and shields the programmer ofapplication program 403 from the internal protocol by which storagesystem 407 communicates. It should be appreciated that the aspects ofthe present invention described herein are not limited to use in asystem that employs an API on the host specifically dedicated tofacilitating communication between the host 401 and the storage system407, as the nature of the host/storage system interface can beimplemented in any of numerous ways, as the present invention is notlimited to any particular implementation technique. As an example of onealternative implementation, application 403 can be customized tocommunicate directly with the storage system 407.

In one embodiment of the invention, storage system 407 may be a contentaddressable storage (CAS) system of the type described in commonlyassigned co-pending applications Ser. No. 09/236,366, entitled “ContentAddressable Information Encapsulation, Representation, and Transfer”,filed Jan. 21, 1999; Ser. No. 09/235,146 entitled “Access to ContentAddressable Data Over A Network”, filed Jan. 21 1999; Ser. No.09/391,360, entitled System and Method for Secure Storage, Transfer andRetrieval of Content Addressable Information, filed Sep. 7, 1999; andthe application entitled “Method And Apparatus For Data Retention In AStorage System”, filed on even date herewith, and not yet assigned aserial number, each of which is incorporated herein by reference. Theseapplications relate to content addressable storage (CAS) and arereferred to collectively as “the CAS applications”.

As discussed above, in a content-addressable system, data is storedusing a content address generated based upon the content of the dataitself. The content address may be generated by applying a hash functionto the data to be stored. The output of the hash function may be used asat least part of the content address used in communication between thehost and storage system to refer to the data. The content address can bemapped (e.g., within the storage system 407) to one or more physicalstorage locations within the storage system.

One example of a hash function that may be used to generate the contentaddress is message digest 5 (MD5). The content address can corresponddirectly to the result of the hash of the content, or additionalinformation can be added to the hash result to generate the address. Itshould be understood that any type of hash function may be used, as theaspects of the invention described herein are not limited to the use ofany particular type of hash function, and some aspects are not evenlimited to use in a content addressable system.

In one embodiment of the invention, a guaranteed unique identifier(GUID) may be added to one or more of the hash results to ensure thatthe content addresses for units of data are unique. The GUID may begenerated, for example, based on the MAC address of the host computerthat generates the content address, a timestamp, and a random number, orin any other suitable manner. One example for generating GUIDs that maybe used in embodiments of the present invention is described in theInternational Standards Organization (ISO) specification ISO/IEC11578:1996.

When a host sends data to the storage system to be stored therein, boththe host and the storage system may independently compute the contentaddress of the data. The host may retain the content address for futureuse to retrieve the data from the storage system. Alternatively, onlythe storage system may compute the content address and return thecomputed content address to the host for later use in accessing thedata. As yet another alternative, both the storage system and the hostmay compute the content address, and the storage system can return itsgenerated content address to the host. The host can compare itsindependently computed content address to the one received from thestorage system to verify that they match.

In the illustrative configuration of FIG. 4, storage system 407 is adistributed storage system including a plurality of access nodes 409 a,409 b, 409 c, and 409 d and a plurality of storage nodes 411 a, 411 b,411 c, and 411 d. The access nodes 409 and storage nodes 407 may becoupled in any suitable way, such as by a communications network (notshown) that employs the TCP/IP network protocol. Each access node 409and storage node 411 may be, for example, a separate computer havingindependent hardware and software resources (i.e., processor orprocessors, memory, disk resources, operating system, storage systemsoftware), although the aspects of the present invention are not limitedin this respect.

As mentioned above, host computer 401 issues access requests (e.g.,reads, writes or other requests) to one of access nodes 409. In the caseof write requests, access nodes 409 may receive the write request (andthe data to be written) from the host computer, select a storage node411 to store the data, and send the data to that storage node forstorage. In some embodiments, access nodes 409 may select a storage node411 in any suitable manner. For example, access nodes 409 may employ aload balancing scheme to distribute data evenly across the storagenodes, or access nodes 409 may employ a different selection technique.As mentioned above in connection with FIG. 2, when storing a unit ofdata, a storage node may create a file having the content address of theunit of data as its filename and may store the unit of data in thatfile.

In the case of read requests, access nodes 409 may receive the requestfrom the host computer, determine on which storage node 411 therequested data is stored, retrieve the requested data from theappropriate storage node 411, and return the requested data to the hostcomputer. The aspects of the present invention described herein are notlimited to this or any other particular configuration, as any suitableconfiguration of the storage system can be employed. For example, ratherthan having the access nodes retrieve the requested data from storagenodes and forward the data to the host 401, the storage system couldalternatively be configured so that the access node simply instructs thestorage node to return the data directly to the host 401.

Storage nodes 411 may receive read and write requests from access nodes409 and, in response, retrieve and store data. As such, storage nodes411 may include one or more storage devices (e.g., disk drives or anyother type of storage device) on which units of data are stored. Itshould be appreciated that access nodes 409 may also include one or morestorage devices, such that access nodes 409 and storage nodes 411 mayhave the same (or approximately the same) or different hardwareconfigurations:

FIG. 4 illustrates a storage system having four access nodes and fourstorage nodes. However, it should be appreciated that this configurationis merely illustrative, as the storage system may include any suitablenumber of access nodes and any suitable number (which may differ fromthe number of access nodes) of storage nodes, as the invention is notlimited to use on a storage system configured in any particular mannerin this respect.

Further, it should be appreciated that while the example of FIG. 4includes access nodes with which host computers communicate to accessthe storage system and storage nodes that store the units of data, theinvention is not limited in this respect. For example, the storagesystem may include nodes which both communicate with host computers andstore units of data. That is, one or more nodes may serve (at least inpart) as both a storage node and access node.

Location Hints

In one embodiment of the invention, the storage system may providelocation hints to the host to increase the efficiency of accesses to thestorage system. A location hint is information provided from the storagesystem to the host and relating to where a particular unit of data isstored on the storage system. The information provided in the locationhint can then be passed back from the host to the storage system inconnection with future access requests for the unit of data, and theinformation can be used by the storage system to efficiently access theunit of data. The location hint for a particular unit of data mayinclude information regarding the storage node on which the unit of datais stored, the physical storage device on that storage node that storesthe unit of data, the file system location of the file containing thatunit of data on that physical storage device, the location on thephysical device of which the unit of data is stored, or any combinationthereof. In addition, as opposed to information that directly identifiesone or more aspects of the physical storage location for the unit ofdata, the information provided in a location hint can be related towhere the unit of data is physically stored in other ways. For example,the storage system could include a index of the physical storagelocation for units of data and the location hint provided to the hostcould provide a key into this index. Thus, the information provided inthe location hint can be related to where the unit of data is physicallystored in any suitable way.

The storage system may provide the location hint to the host in any oneof a variety of ways, as the invention is not limited in this respect.For example, in response to a write request from the host computer towrite a unit of data, the storage system may return the location hintinformation (e.g., as a token or in any form) for the unit of data. Thelocation hint may be the only information returned, or it may be sentback to the host with other information returned to the host in responseto a write request. For example, in one embodiment, the storage systemresponds to a write request by returning the content address (e.g., theMD5 hash value) corresponding to the unit of data so that the host mayverify that the storage system has computed the content addresscorrectly. In one embodiment, when returning the content address to thehost in response to the write request, the storage system may also sendthe location hint, as shown conceptually at 406 in FIG. 4.

The storage system may also provide location hint information to thehost in response to a read request. For example, when the storage systemreturns the requested unit of data to the host, the storage system mayalso send the location hint information for the requested unit of data.

The storage system may also provide location hint information to thehost in response to query requests, or any other type of request. Aquery request is a request to the storage system in which a host mayrequest units of data based on information about the units of data. Suchinformation may be, for example, the time at which the units of datawere stored on the storage system, part of the metadata stored in a CDF,a partial content address, or any other suitable information.

For example, instead of providing a content address to the storagesystem (as in a typical read request), the host may provide a time range(e.g., a start time and an end time) to the storage system. In response,the storage system returns to the host a list of the content addressescorresponding to units of data that were stored on the storage systemduring the period of time indicated by the time range. The storagesystem may send location hint information to the host for each of theunits identified in response to the query request. Providing locationhints in response to a query request may be useful because it is likelythat the host will issue read requests for at least some of the contentaddresses returned by the storage system in response to the queryrequest. If the host can inform the storage system of the location ofthe units of data corresponding to these content addresses (i.e., fromthe location hint), computational resources (and time) on the storagesystem expended in locating the units of data on the storage system canbe reduced.

As mentioned above, the information that may be provided to the storagesystem in a query request is not limited to a time range. Theinformation may additionally or alternatively include part of themetadata stored in a CDF, a partial content address, or any othersuitable information. For example, if the query request includesmetadata stored in a CDF or a partial content address, the storagesystem may return the content addresses of any CDFs that include thespecified metadata or partial content address.

The host may use the location hint information received from the storagesystem in any of a variety of ways, as the invention is not limited inthis respect. As mentioned above, an application program 403 (FIG. 4)executing on the host may communicate with the storage system through anAPI 405. When the API receives a location hint from the storage system(e.g., in response to a read, write, query request, or other accessrequest), the API may provide the location hint to the applicationprogram. The application program may then store the location hintinformation in a manner that associates it with the correspondingcontent address. Thus, the next time the application program requestsaccess to the unit of data corresponding to that content address (i.e.,through the API), the application program may provide the location hintinformation along with the content address.

Alternatively, instead of passing the location hint information to theapplication program, the API may store the location hint informationitself. Thus, the API may receive from the application program a requestto access a unit of data on the storage system, with the requestincluding the content address for the desired unit of data. The API maydetermine if it has stored a location hint corresponding to that contentaddress. If so, the API may provide the location hint to the storagesystem along with the access request.

The host (e.g., the API or the application program) may store thelocation hint information in any suitable manner. In one embodiment, thehost (e.g., the API) maintains a cache of stored location hints andtheir corresponding content addresses for performance reasons. When thecache is full, the API or application program may replace items in thecache using a least-recently used replacement scheme, most-recently usedreplacement scheme, or any other suitable cache replacement scheme.

It should be appreciated that the use of location hints shifts some ofthe computational burden of locating data on the storage system from thestorage system to the host, as the host computer maintains and accessesrecords of location information. When issuing an access request to thestorage system, the host computer may search these records and providelocation information, if available, to the storage system. This enablesthe storage system to avoid at least some of the searching of records onthe storage system to determine the physical location for a stored dataunit.

In another embodiment, the host may simply ignore any location hintsthat it receives from the storage system. That is, the host may choosenot to bear any of the computational burden of locating data and mayrely on the storage system to locate the content.

In another embodiment, the use of location hints is provided as anoptional feature to the host, such that the feature can be turned on oroff.

It should be appreciated that the location hint information can bedetermined by the storage system in any of numerous ways, as the presentinvention is not limited to any particular implementation technique. Inaccordance with one embodiment of the present invention for use with astorage system having the configuration shown in FIG. 4, the locationinformation can be generated by the access nodes 409 and possibly alsoby the storage nodes 411 depending upon the granularity of the locationinformation provided. In this respect, and as discussed above, thelocation information can be provided in any of numerous levels ofgranularity, as the present invention is not limited in this respect.For example, the location information can provide a hint only as towhich access node controls access to the physical storage location, orcan provide additional information relating to a specific storage nodethat stores the unit of data. Similarly, the information relating to theparticular storage node that stores the unit of data can also beprovided at numerous levels of granularity, as this information canmerely identify the storage node, or can alternatively be moredescriptive and can identify a file system location and optionally alsoa particular storage device (e.g., disk drive) on which the unit of datais stored. It should be appreciated that in response to an accessrequest, it is a fairly simple process for the storage system togenerate the location information, because, as of a result of accessingthe unit of data, the storage system will go through the process ofdetermining the physical storage location and will have this informationavailable to provide in a hint to the host. For example, in oneembodiment of the present invention, when a unit of data is accessed,the storage node on which the data is stored identifies to theappropriate access node the file system and disk drive on which the datais stored, and the access node then passes this information along to thehost in the form of the location hint described above. It should beappreciated that the levels of granularity and the manner in which thelocation information is generated is not limited to the examplesdescribed herein, nor to a storage system having the type ofconfiguration shown in FIG. 4, as this aspect of the present inventioncan be used in connection with numerous other types of storage systemconfigurations, including those that do not employ an access node andstorage node structure.

If location hints are not used (i.e., the feature is turned off), or ifthe host does not have location information for a particular unit ofdata, the storage system may use a multi-cast location query, BLIlookup, or any other suitable technique to locate the data. Further, insome circumstances, a location hint may not correctly identify thelocation of the content on the storage system. Such a situation mayoccur, for example, when the storage system moves the content to a newlocation after the host has received the location hint. If a locationhint fails to identify the correct location of the content on thestorage system, the storage system may use another method of locatingthe content, such as MLQ, BLI lookup, or any other suitable technique.

BLI Caching

In one embodiment of the invention, the storage system maintains a dataset (e.g., a database) referred to as a blob location index (BLI) thatworks in the manner of the BLI described above in connection with aknown system. In one embodiment, the responsibility of administering theBLI is distributed among storage nodes 411 (FIG. 4), with each having adesignated range of addresses (e.g., content addresses) that it isresponsible for and an index for the addresses in that range to identifywhere the corresponding data units are stored. The BLI itself (inaddition to the administration responsibilities) may also be distributedacross multiple nodes of the storage system. To increase the efficiencyof accesses to the BLI, portions of the BLI are cached in accordancewith one embodiment of the invention.

The blob location index may be a database (or other data set) that maps,for units of data stored on the storage system, the content addresscorresponding to the unit of data to the storage node on which the unitof data is stored. The BLI may be stored in its entirety on one storagenode (and optionally mirrored on one or more other storage nodes) or maybe distributed among two or more storage nodes as discussed above. Inresponse to an access request to a particular address, the access node409 that receives the request directs it to a storage node 411 thatadministers the portion of the BLI that includes that address, and theBLI is read by the appropriate storage node to determine the location ofthe requested unit of data on the storage system.

It should be understood that the administration of a single portion ofthe BLI may be shared by more than one storage node, in which case theaccess node 409 may direct such requests to one or all of the storagenodes that administer that portion of the BLI.

It should also be understood that in some embodiments of the invention,storage of the BLI or administration of the BLI may be performed by bothaccess nodes and storage nodes, or solely by access nodes, as theinvention is not limited in this respect. It should further beappreciated that the techniques discussed below for caching portions ofthe BLI are equally applicable to both access nodes and storage nodes.

In the prior art system, the BLI is stored in secondary memory (e.g.,magnetic disk), which is slower to access than primary memory. When theaccess request is a write request for a new unit of data with no entryon the BLI, the BLI is updated to include the content address andstorage node location of the newly stored data. The storage system waitsfor the BLI to be updated before indicating to the host that the writehas been successfully completed. If the portion of the BLI to be updatedis not already in the primary memory of the storage node, the storagenode updating the BLI performs a disk read to read the BLI into primarymemory (i.e., RAM), updates the BLI in primary memory, and then performsa disk write to write the updated BLI back to disk. Thus, from the pointof view of an application program, there is some latency from the time awrite request is issued to the time that the storage system confirms asuccessful write and returns the content address of the written data,and this latency is increased when the confirmation is not returneduntil the BLI is updated.

In accordance with one embodiment of the invention, the storage systemmaintains a BLI cache in primary memory (e.g., RAM) to provide forimproved access performance. For example, in response to a writerequest, the storage system can write the new entry into the BLI cachein primary memory and respond to the requesting host that the write hascompleted successfully. The contents of the BLI cache can be destaged tothe BLI on disk at a later time. As a result, the latency caused byupdating the BLI is reduced.

The BLI cache also increases the efficiency of the read accesses to thestorage system. When a read request is received, the storage system maysearch the BLI cache to determine if location of the requested unit ofdata is stored in the BLI cache. If it is, the need for a disk read tothe BLI is obviated.

The BLI cache can be arranged and stored in any suitable manner as thepresent invention is not limited to any particular implementationtechnique. For example, when the BLI is stored entirely on a singlestorage node, the BLI cache may be maintained entirely on the samestorage node or distributed among multiple storage nodes. Similarly,when the BLI is distributed across multiple storage nodes, one storagenode may be used to store the BLI cache or each storage node could storethe portion of the BLI cache (412 a-d in FIG. 4) corresponding to theportion of the BLI stored by that storage node.

It should be understood that the above described locations of the BLIcache are given only as examples, as the BLI cache may be locatedanywhere in the storage system and/or distributed across the storagesystem in any manner, with entries of a distributed BLI cache notnecessarily being limited to being on the same machine as correspondingentries for the BLI. It should further be appreciated that the BLI cachemay be destaged to the BLI in any suitable manner, as the invention isnot limited in this respect. For example, entries in the BLI cache maybe destaged to the BLI whenever the BLI cache is full according to anysuitable replacement algorithm. Alternatively, BLI cache entries may bedestaged to the BLI during periods of light processor usage within thestorage system. As yet another alternative, BLI cache entries may bedestaged to the BLI after a fixed period of time has elapsed.

In one embodiment, access nodes may also cache BLI entries, eitherinstead of or in addition to storing BLI cache entries on the storagenodes. When an access node processes a write request from a host, theaccess node selects on which storage node the data is written, so theaccess node has this information along with the content address of thenewly written data and can add it in its BLI cache. In one embodiment,after the storage node stores the newly written data, it can returninformation to the access node regarding the file system and disk on thestorage node where the data is stored so that this information also canbe added to the cache entry on the access node. It should be appreciatedthat the present invention is not limited in this respect, as the BLIcache entries can, alternatively, identify only the storage node onwhich the data units are stored.

When the access node receives a read request, it may search for therequested content address in its BLI cache. If there is a cache hit(i.e., the content address is found in the cache), then the access nodeneed not query the BLI on the storage system, as the access nodedetermines the storage location for the requested unit of data from itsown BLI cache. Thus, the response time in returning the requested unitof data may be further reduced.

It should be appreciated that the BLI cache on the access nodes can beused with or without another layer of BLI caching on the storage nodes.It should also be understood that the BLI, as described above, mapscontent addresses to storage nodes which store the units of datacorresponding to those content addresses. However, the BLI may alsoinclude information about the particular physical disk device and/orfile system location of the unit of data, as the invention is notlimited in this respect.

In one embodiment, BLI caching is provided as an optional feature to theuser of the storage system, such that the feature can be turned on oroff.

As discussed above, when a new unit of data is written to the storagesystem, in addition to writing the data to a storage node and updatingthe metadata in the storage system that relates to where the unit ofdata is stored, in the embodiment that employs a BLI cache, the BLIcache may be updated as well. As further mentioned above, the updatingof numerous areas in the storage system may provide some latency interms of when a confirmation is sent from the storage system to the hostthat a write request has been processed. In accordance with oneembodiment of the present invention, the write to the BLI cache can bedone after the confirmation is provided to the host, as the failure tohave an entry in the BLI cache for a unit of data will only impact theperformance of a future access to the unit of data, but will notpreclude the unit of data from being accessed by the storage system. Ina further embodiment of the present invention, the confirmation can alsobe provided even before any appropriate entries are written to the BLI,to further improve write performance. In this respect, it should beappreciated that in accordance with one embodiment of the presentinvention, if an access request is received to read data and nocorresponding entry is provided in the BLI, the storage system may issuea multicast location query for the unit of data to see whether it isstored on any of the storage nodes. Thus, in accordance with thisembodiment of the present invention, the storage system is able tolocate units of data even if they do not include a corresponding entryin the BLI.

It should be appreciated that the above discussion relating toimplementing a cache of a location index in a content-addressablestorage system having the configuration shown in FIG. 4 is providedmerely for illustrative purposes, and that the present invention is notlimited in this respect. The aspect of the present invention relating tothe use of a cache for a location index in a content-addressable storagesystem can be employed with storage systems having configurations otherthan the access node/storage node configuration shown in FIG. 4.

Furthermore, it should be appreciated that the aspects of the presentinvention relating to caching a location index and the use of a locationhint that can be passed to the host computer are not mutually exclusive,and that these two features can be used together as well as separately.

Intrinsic Locations

An alternate embodiment uses a technique (referred to herein asintrinsic locations) that employs at least a portion of the contentaddress to determine at least one aspect of the storage location forunits of data on the storage system. FIG. 5 shows an illustrativestorage system 507 using intrinsic locations. Storage system 507includes a plurality of access nodes 503 a, 503 b, and 503 c and aplurality of storage nodes 505 a, 505 b, 505 c, 505 d, 505 e, and 505 f.The storage nodes may be divided into groups such that each group ofstorage nodes is assigned a range of content address. In the example ofFIG. 5, storage nodes 505 a and 505 b are assigned content addressesbeginning with characters ‘A’-‘I’, storage nodes 505 c and 505 d areassigned content addresses beginning with characters ‘J’-‘R’, andstorage nodes 505 e and 505 f are assigned content addresses beginningwith characters ‘S’-‘Z’. In FIG. 5, each group of storage nodes includestwo storages nodes. However, it should be understood that a storage nodegroup may include only one storage node or may include three or morestorage nodes. Further, it is not necessary that each group of storagenodes include an equal number of storage nodes.

In one embodiment, each group of storage nodes stores the data unitscorresponding to its assigned range of content addresses. Access nodes503 maintain information of which range of content addresses areassigned to each group of storage nodes and, as such, may select theappropriate storage node to access units of data in response to anaccess request.

For example, if access node 503 a receives from host 501 a a request towrite a unit of data whose content address begins with the character‘J’, access node will select either storage node 505 c or 505 d to storethe data. In one embodiment, access node 503 a selects one of these twostorage nodes based on another part of the content address. For example,access node 503 a may examine the second character of the contentaddress to select the appropriate storage node within the selectedgroup. For example, storage node 505 c may store data units whosecontent addresses have a second character of ‘A’-‘M’ and storage node505 d may store data units whose content addresses have a secondcharacter of ‘N’-‘Z’. It should be appreciated that any character(s) inthe content address may be used to select the appropriate storage nodewithin a group of multiple storage nodes, as the invention is notlimited in this respect.

Once access node 503 a has selected a storage node to store the unit ofdata, the selected storage node may select a physical storage device(e.g., disk) on which to store the unit of data. FIG. 6 conceptuallyshows a storage node 601 having four physical storage devices, 603 a,603 b, 603 c, and 603 d. The storage node 601 may select one of thesestorage devices to store a unit of data based on one or more charactersin the content address. For example, storage node 601 may select one ofthe storage devices 603 based on the third character of the contentaddress. If the third character of the content address is ‘A’-‘F’,storage device 603 a may store the data. If the third character is‘G’-‘L’, storage device 603 b may store the data. If the third characteris ‘M’-‘R’, storage device 603 c stores the data. If the third characteris ‘S’-‘Z’ storage device 603 d stores the data.

By selecting the physical storage location (e.g., storage node andstorage device) of a unit of data based on the content address of theunit of data, an access node may later determine the physical locationwithout a database lookup or multicast location query. That is, theaccess node may simply examine the content address and determine thelocation of the unit of data based on the content address.

While the illustrative example described above uses the intrinsiclocation technique to determine not only the storage node on which aunit of data is stored based upon its content address, but also the filesystem and storage device within that node, it should be appreciatedthat the aspect of the present invention relating to the use ofintrinsic locations is not limited in this respect, as it can beemployed to determine only some aspects of the physical storage locationof a unit of data but not others. For example, the intrinsic locationstechnique could alternatively be used to select only a storage node onwhich a unit of data is stored, and to thereby eliminate the need toperform lookups or broadcast queries to determine the storage node, butthe determination of the file system and/or physical storage device onwhich the unit of data is stored can be performed in other ways,including through the use of indexing techniques as described above.

Applicants have appreciated that for some units of data, it may bedesirable to not store them in the particular physical storage locationspecified for the unit of data by the intrinsic locations techniquebased upon its content address, and Applicants have developed atechnique to address such units of data. This can occur for any ofnumerous reasons, an example of which is a particularly large unit ofdata. For a large unit of data, there may not be room on the physicalstorage location specified by the mapping performed by the intrinsiclocations technique to store the data, or it may be preferable to storeit elsewhere for load balancing purposes. In addition, some applicationprograms that write a particularly large unit of data to a storagesystem may wish to begin the write operation before generating a contentaddress based upon the entire unit of data, such that the writeoperation may begin for a unit a data before the content address hasbeen determined. Thus, when the intrinsic locations technique isemployed, there may not be a content address to determine where to storethe unit of data when the unit of data is passed to the storage system.In accordance with one embodiment of the present invention, theabove-described situations can be addressed by using the intrinsiclocations technique and pointers. For example, if it is desired to storea unit of data (e.g., a relatively large one) in a location other thanthat specified by the intrinsic location mapping technique describedabove, a pointer can be stored in the storage location specified by thecontent address for the unit of data, with the pointer pointing toanother physical storage location at which it is more desirable to storethe unit of data. Thus, when future access requests are received for theunit of data, the intrinsic location mapping techniques will map to thepointer for the unit of data, which can then be read and used toretrieve the data.

The pointer technique can also be employed when handling a write requestfor a unit of data for which no content address has yet been generated.In this respect, when the unit of data is received, the storage locationcan place it in any suitable storage location, and then once the contentaddress for the unit of data is determined and/or provided, a pointercan be placed in the appropriate location determined by the intrinsiclocation mapping technique for the newly provided content address, withthe pointer pointing to the storage location that was chosen for theunit of data. Of course, it should be appreciated that the data couldalternatively be moved to the storage location specified by theintrinsic location mapping technique once the content address isprovided.

Further, in one embodiment, in which the pointer technique is used tostore data, the storage system may execute a background process thatperiodically searches the storage system for pointers pointing to otherstorage locations at which units of data are stored. If any such pointeris found, the process may move the unit of data from the storagelocation pointed to by the pointer to the storage location specified bythe intrinsic location mapping (i.e., the location of the pointer),thereby replacing the pointer with the corresponding unit of data. Thus,the storage system's efficiency in accessing the units of data (e.g., inresponse to a read request from a host computer) may be increased, asthe storage system need not access both the storage location of thepointer and the storage location of the unit of data.

It should be understood that the background process for replacingpointers with their corresponding units of data may be executed on anythe storage system in any suitable way. For example, the process may beexecuted only on storage nodes that are idle. Alternatively, the processmay be executed on every storage node after a predetermined period oftime has elapsed. Many other alternatives are available and theinvention is not limited in this respect.

In accordance with one embodiment of the present invention, if a unit ofdata cannot be found using the intrinsic location technique, alternativelook up techniques can be employed, such as performing a multicastlocation query.

Temporal Locality of Reference

As mentioned above, in one embodiment, units of data are stored in fileson a file system of the corresponding storage node. Such files may havea filename corresponding, at least in part, to the content address ofthe unit of data stored therein, and the file system may be organized indirectories and subdirectories with the name of the subdirectory inwhich a particular unit of data is stored corresponding at least in partto at least a portion of the filename of the unit of data. As mentionedabove, the file system may be organized based on the first characters ofthe content addresses of the units of data stored in the file system.However, it should be appreciated that the invention is not limited inthis respect, as the file system may be organized using any portion ofthe content address including, but not limited to, the first letters orbits, the middle portion, or the end portion. The hashing algorithm usedto generate content addresses may yield fairly random values (as is thecase with the MD5 hashing algorithm). As a result Applicants haveappreciated that units of data written at approximately the same timemay be stored in different locations in the file system hierarchy. Thatis, despite the fact that the units of data are written in closeproximity in time, these units of data may be scattered across the filesystem. Thus, the randomness of the hashing algorithm precludes anylocality of reference in the file system.

Operating systems typically maintain a file system map that maps filesystem locations to physical disk locations. When the file system islarge (i.e., a large hierarchy of directories and subdirectories) andthere are many files stored in the file system, the file system mapmaintained by the operating system may also be quite large. As a result,the operating system may only keep a small portion of the file systemmap in the primary memory (e.g., RAM) of the storage node at one time.The rest of the file system map may remain in secondary memory (e.g.,disk). As various portions of the file system map are needed (i.e., whenfiles are accessed), they may be brought into primary memory, with otherportions being swapped out to disk. It should be appreciated that accessperformance is better when accessing portions of the file system inmemory rather than disk.

If two units of data are written to the storage node proximate in timebut stored in different portions of the file system map, the operatingsystem first must load from disk the portion of the file system maphaving the file system location of the first unit of data (i.e., todetermine the physical disk location of that unit of data) and then mustreturn to disk to load the portion of the file system map having thefile system location of the second unit of data. These repeated diskaccesses slow the write response time of the storage system. Inaddition, since units of data written proximate in time are also oftenread proximate in time, a similar access delay may be encountered whenreading the data units.

Applicants have appreciated that in a storage system that employs a filesystem structure, it is desirable to store units of data that are storedat approximately the same time in the approximately the same file systemlocation, to minimize the likelihood of the operating system having togo to disk to load different portions of the file system forsequentially accessed data units. As mentioned above, in one embodimentof the invention, the file system location of a file is determined basedon the filename of the file. Thus, in accordance with one embodiment ofthe invention, information is added to the filename of the file that issimilar for files written at approximately the same time. This providestemporal locality of reference for files written at approximately thesame time, and tends to result in those filed being stored in proximatefile system locations.

It should be appreciated that in embodiments of the invention thatemploy content addressable storage, the information that is added to thefilename of the file may also be added to the content address of thedata stored in the file. For example, the information may be added tothe hash of the content of the file to generate a content address basedon the hash and the additional information. The information may be addedat any suitable position in the hash. For example, the information maybe added to the beginning, the end, or at any other position in thehash. The host computer may use the content address to access the filein the manner described above.

In one embodiment of the invention, the information added to establishtemporal locality of reference may be added to the content address atthe storage system. That is, for example, when a unit of data is storedto the storage system, the storage system may generate the temporallocality of reference information, include the information in the hash,and return the resulting content address to the host computer thatsubmitted the storage request.

In an alternate embodiment, the host computer generates the temporallocality of reference information and adds the information to the hashto generate the content address. The host computer may send the contentaddress to the storage system or may just send the temporal locality ofreference information to the storage system and allow the storage systemto add the information to the hash to determine the content address. Bycomputing the temporal locality of reference information at the hostcomputer, if the host computer stores the unit of data on multiplestorage systems at the same time (or approximately the same time), e.g.,for backup or replication purposes, the content address for the unit ofdata is the same on each of the storage systems.

In another embodiment, temporal locality of reference information may begenerated by the storage system and the host computer, and both sets ofinformation may be added to the hash, at any suitable position in thehash, to generate the content address of the unit of data.

The above example was described as providing temporal locality ofreference with respect to files stored in a file system. However itshould be appreciated that such a technique is applicable to otherlogical data storage constructs. For example, adding temporal localityof reference information to content addresses (or other informationstored in a database) may increase access efficiency of database tablesthat are accessed using a content address (or other information) as akey, and can also be used with other logical constructs.

The information added to the filename may be any information that issimilar for data units written at approximately the same time. Forexample, a timestamp may be added to the file names, and the timestampcan be examined by the file system in determining where to store thedata units. Alternatively, a counter value may be added to the filenames with the counter value being incremented each time a new data unitis written to the storage system. Again, any suitable information may beused, as the invention is not limited in this respect.

It should be appreciated that the aspect of the present invention thatrelates to adding information to a content address that provides anindication of which units of data are written to the storage systemproximate in time is not limited to a storage system having theparticular access node and storage node configuration described, or toemploying a file system on the storage node as described above, as thisaspect of the present invention can be used in connection with storagesystems having numerous different configurations.

FIG. 7 conceptually illustrates the temporal locality of referenceaspect of the present invention, as it illustrates locality information703 being added to a content address 701. It should be appreciated thatthis aspect of the present invention is not limited to providinglocality information at the beginning of the content address as shown inFIG. 7, as the locality information can be added to the content addressin any suitable manner, including providing it at the end of the contentaddress, or within the range of the content address, and includingproviding locality information as a contiguous block as shown in FIG. 7,or dispersed throughout the content address information 701.

The above-described embodiments of the present invention can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. It should beappreciated that any component or collection of components that performthe functions described above can be generically considered as one ormore controllers that control the above-discussed functions. The one ormore controllers can be implemented in numerous ways, such as withdedicated hardware, or with general purpose hardware (e.g., one or moreprocessors) that is programmed using microcode or software to performthe functions recited above.

In this respect, it should be appreciated that one implementation of theembodiments of the present invention comprises at least onecomputer-readable medium (e.g., a computer memory, a floppy disk, acompact disk, a tape, etc.) encoded with a computer program (i.e., aplurality of instructions), which, when executed on a processor,performs the above-discussed functions of the embodiments of the presentinvention. The computer-readable medium can be transportable such thatthe program stored thereon can be loaded onto any computer systemresource to implement the aspects of the present invention discussedherein. In addition, it should be appreciated that the reference to acomputer program which, when executed, performs the above-discussedfunctions, is not limited to an application program running on a hostcomputer. Rather, the term computer program is used herein in a genericsense to reference any type of computer code (e.g., software ormicrocode) that can be employed to program a processor to implement theabove-discussed aspects of the present invention.

It should be appreciated that in accordance with several embodiments ofthe present invention wherein processes are implemented in a computerreadable medium, the computer implemented processes may, during thecourse of their execution, receive input manually (e.g., from a user).

In various examples described above, content addresses were described toinclude alphabetic characters ‘A’-‘Z’. It should be understood thatthese content addresses were given only as examples, and that contentaddresses may include any alphanumeric character, series of bits, or anyother suitable character, as the invention is not limited in thisrespect.

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having,” “containing”, “involving”, andvariations thereof herein, is meant to encompass the items listedthereafter and additional items.

Having described several embodiments of the invention in detail, variousmodifications and improvements will readily occur to those skilled inthe art. Such modifications and improvements are intended to be withinthe spirit and scope of the invention. Accordingly, the foregoingdescription is by way of example only, and is not intended as limiting.The invention is limited only as defined by the following claims and theequivalents thereto.

1. A method of processing data in a computer system comprising at leastone host and at least one storage system which stores data for the atleast one host, the method comprising acts of: (a) receiving, at thestorage system, a first request from the at least one host to access aunit of data stored on the at least one storage system; (b) in responseto the first request, returning to the at least one host informationrelated to where the unit of data is physically stored on the at leastone storage system; and (c) subsequent to the act (b), receiving, at theat least one storage system, a second request to access the unit ofdata, the second request including the information related to where theunit of data is physically stored on the storage system.
 2. The methodof claim 1, wherein the storage system is a content addressable storagesystem and the at least one host accesses data units stored on the atleast one storage system using content addresses generated based on thecontent of the data units, and wherein the first request received in theact (a) identifies the unit of data using a content address.
 3. Themethod of claim 1, further comprising an act of using the informationrelated to where the unit of data is physically stored on the storagesystem received in the act (c) to retrieve the unit of data.
 4. Themethod of claim 1, wherein the act (a) further comprises an act ofreceiving a query request that specifies at least one criterion andrequests units of data stored on the storage system that satisfy the atleast one criterion.
 5. The method of claim 4, wherein the at least onecriterion includes a time period in which units of data were stored onthe storage system.
 6. The method of claim 1, wherein the first requestto access the unit of data includes a request to write the unit of data,and wherein the act (a) further comprises an act of receiving the unitof data from the host.
 7. The method of claim 6, further comprising anact of returning, to the host, an address associated with the unit ofdata.
 8. The method of claim 7, further comprising an act of using theinformation related to where the unit of data is physically stored onthe storage system received in the act (c) to retrieve the unit of data.9. The method of claim 1, wherein the storage system includes aplurality of storage nodes, and wherein the information related to wherethe unit of data is physically stored on the storage system includesinformation identifying the ones of the plurality of storage nodes onwhich the unit of data is stored.
 10. The method of claim 9, wherein atleast one of the plurality of storage nodes includes a plurality ofdisks, and wherein the information related to where the unit of data isphysically stored on the storage system includes information related toon which of the plurality of disks the unit of data is stored.
 11. Atleast one computer readable medium encoded with instructions that, whenexecuted on a computer system perform a method of processing data,wherein the computer system comprises at least one host and at least onestorage system which stores data for the at least one host, the methodcomprising acts of: (a) receiving, at the storage system, a firstrequest from the at least one host to access a unit of data stored onthe at least one storage system; (b) in response to the first request,returning to the at least one host information related to where the unitof data is physically stored on the at least one storage system; and (c)subsequent to the act (b), receiving, at the at least one storagesystem, a second request to access the unit of data, the second requestincluding the information related to where the unit of data isphysically stored on the storage system.
 12. The at least one computerreadable medium of claim 11, wherein the storage system is a contentaddressable storage system and the at least one host accesses data unitsstored on the at least one storage system using content addressesgenerated based on the content of the data units, and wherein the firstrequest received in the act (a) identifies the unit of data using acontent address.
 13. The at least one computer readable medium of claim11, wherein the method further comprises an act of using the informationrelated to where the unit of data is physically stored on the storagesystem received in the act (c) to retrieve the unit of data.
 14. The atleast one computer readable medium of claim 12, wherein the act (a)further comprises an act of receiving a query request that specifies atleast one criterion and requests units of data stored on the storagesystem that satisfy the at least one criterion.
 15. The at least onecomputer readable medium of claim 14, wherein the at least one criterionincludes a time period in which units of data were stored on the storagesystem.
 16. The at least one computer readable medium of claim 11,wherein the first request to access the unit of data includes a requestto write the unit of data, and wherein the act (a) further comprises anact of receiving the unit of data from the host.
 17. The at least onecomputer readable medium of claim of claim 11, wherein the methodfurther comprises an act of returning, to the host, an addressassociated with the unit of data.
 18. The at least one computer readablemedium of claim 17, wherein the method further comprises an act of usingthe information related to where the unit of data is physically storedon the storage system received in the act (c) to retrieve the unit ofdata.
 19. The at least one computer readable medium of claim 11, whereinthe storage system includes a plurality of storage nodes, and whereinthe information related to where the unit of data is physically storedon the storage system includes information identifying the ones of theplurality of storage nodes on which the unit of data is stored.
 20. Theat least one computer readable medium of claim 19, wherein at least oneof the plurality of storage nodes includes a plurality of disks, andwherein the information related to where the unit of data is physicallystored on the storage system includes information related to on which ofthe plurality of disks the unit of data is stored.
 21. A storage systemfor use in a computer system that includes at least one host, whereinthe storage system stores data for the at least one host, the storagesystem comprising: at least one storage device to store data receivedfrom the at least one host; and at least one controller that receives afirst request from the at least one host to access a unit of data storedon the storage system and in response to the first request, returns tothe at least one host information related to where the unit of data isphysically stored on the storage system; wherein the at least onecontroller, subsequent to returning to the at least one host informationrelated to where the unit of data is physically stored, receives asecond request to access the unit of data, the second request includingthe information related to where the unit of data is physically storedon the storage system.
 22. The storage system of claim 21, wherein thestorage system is a content addressable storage system, wherein the atleast one host accesses data units stored on the at least one storagesystem using content addresses generated based on the content of thedata units, and wherein the first request from the at least one hostidentifies the unit of data using a content address.
 23. The storagesystem of claim 21, wherein the at least one controller uses theinformation received in the second request to retrieve the unit of data.24. The storage system of claim 21, wherein the at least one controllerreceives a query request that specifies at least one criterion andrequests units of data stored on the storage system that satisfy the atleast one criterion, and wherein the at least one controller processesthe query request.
 25. The storage system of claim 24, wherein the atleast one criterion includes a time period in which units of data werestored on the storage system.
 26. The storage system of claim 21,wherein the first request to access the unit of data includes a requestto write the unit of data, and wherein the at least one controllerreceives the unit of data from the at least one host.
 27. The storagesystem of claim 26, wherein the at least one controller, subsequent toreturning to the at least one host information related to where the unitof data is physically stored, returns to the at least one host anaddress associated with the unit of data.
 28. The storage system ofclaim 27, wherein the at least one controller uses the informationreceived in the second request to retrieve the unit of data.
 29. Thestorage system of claim 21, further comprising a plurality of storagenodes, wherein the information related to where the unit of data isphysically stored on the content addressable storage system includesinformation identifying the ones of the plurality of storage nodes onwhich the unit of data is stored.
 30. The storage system of claim 29,wherein at least one of the plurality of storage nodes includes aplurality of disks, and wherein the information related to where theunit of data is physically stored on the content addressable storagesystem includes information related to on which of the plurality ofdisks the unit of data is stored.
 31. A method of processing data in acomputer system comprising at least one host and at least one storagesystem which stores data for the at least one host, the methodcomprising acts of: (a) sending, to the at least one storage system, afirst request to access a unit of data stored on the storage system; (b)receiving, from the at least one storage system, information related towhere the unit of data is physically stored on the at least one storagesystem; and (c) subsequent to the act (b), sending to the at least onestorage system a second request to access the unit of data, the secondrequest including the information related to where the unit of data isphysically stored on the at least one storage system.
 32. The method ofclaim 31, wherein the at least one storage system is a contentaddressable storage system and the at least one host accesses data unitsstored on the at least one storage system using content addressesgenerated based on the content of the data units, and wherein the firstrequest sent in the act (a) identifies the unit of data using a contentaddress.
 33. The method of claim 31, wherein the act (a) furthercomprises an act of sending, to the at least one storage system, a queryrequest that specifies at least one criterion and requests units of datastored on the storage system that satisfy the at least one criterion.34. The method of claim 33, wherein the at least one criterion includesa time period in which units of data were stored on the storage system.35. The method of claim 31, wherein the first request to access the unitof data includes a request to write the unit of data, and wherein theact (a) further comprises an act of sending the unit of data to the atleast one storage system.
 36. The method of claim 31, wherein the firstrequest to access the unit of data includes a request to read the unitof data, and wherein the method further comprises an act of receivingthe unit of data from the at least one storage system.
 37. The method ofclaim 36, wherein the act (a) further comprises an act of sending, tothe at least one storage system, an address associated with the unit ofdata.
 38. At least one computer readable medium encoded withinstructions that, when executed on a computer system, perform a methodof processing data, wherein the computer system comprises at least onehost and at least one storage system which stores data for the at leastone host, the method comprising acts of: (a) sending, to the at leastone storage system, a first request to access a unit of data stored onthe storage system; and (b) receiving, from the at least one storagesystem, information related to where the unit of data is physicallystored on the at least one storage system; and (c) subsequent to the act(b), sending to the at least one storage system a second request toaccess the unit of data, the second request including the informationrelated to where the unit of data is physically stored on the at leastone storage system.
 39. The at least one computer readable medium ofclaim 38, wherein the at least one storage system is a contentaddressable storage system and the at least one host accesses data unitsstored on the at least one storage system using content addressesgenerated based on the content of the data units, and wherein the firstrequest sent in the act (a) identifies the unit of data using a contentaddress.
 40. The at least one computer readable medium of claim 38,wherein the act (a) further comprises an act of sending, to the at leastone storage system, a query request that specifies at least onecriterion and requests units of data stored on the storage system thatsatisfy the at least one criterion.
 41. The at least one computerreadable medium of claim 40, wherein the at least one criterion includesa time period in which units of data were stored on the storage system.42. The at least one computer readable medium of claim 38, wherein thefirst request to access the unit of data includes a request to write theunit of data, and wherein the act (a) further comprises an act ofsending the unit of data to the at least one storage system.
 43. The atleast one computer readable medium of claim 38, wherein the firstrequest to access the unit of data includes a request to read the unitof data, and wherein the method further comprises an act of receivingthe unit of data from the at least one storage system.
 44. The at leastone computer readable medium of claim 43, wherein the act (a) furthercomprises an act of sending, to the at least one storage system, anaddress associated with the unit of data.
 45. A host computer for use ina computer system that includes the host computer and at least onestorage system, the host computer comprising: at least one storagedevice; and at least one controller that: sends, to the at least onestorage system, a first request to access a unit of data stored on theat least one storage system; and in response to the first request,receives from the at least one storage system, information related towhere the unit of data is physically stored on the at least one storagesystem and stores the information in the at least one storage device;wherein the at least one controller, subsequent to receiving theinformation related to where the unit of data is physically stored,sends, to the at least one storage system, a second request to accessthe unit of data, the second request including the information relatedto where the unit of data is physically stored on the at least onestorage system.
 46. The host computer of claim 45, wherein the at leastone storage system is a content addressable storage system and the atleast one host accesses data units stored on the at least one storagesystem using content addresses generated based on the content of thedata units, and wherein the request sent to the at least one storagesystem identifies the unit of data using a content address.
 47. The hostcomputer of claim 45, wherein the first request to access the unit ofdata includes a query request that specifies at least one criterion andrequests units of data stored on the storage system that satisfy the atleast one criterion.
 48. The host computer of claim 47, wherein the atleast one criterion includes a time period in which units of data werestored on the storage system.
 49. The host computer of claim 45, whereinthe first request to access the unit of data includes a request to writethe unit of data, and wherein the at least one controller sends the unitof data to the at least one storage system.
 50. The host computer ofclaim 45, wherein the first request to access the unit of data includesa request to read the unit of data, and wherein the at least onecontroller sends the unit of data to the at least one storage system.51. The host computer of claim 50, wherein the at least one controllersends to the at least one storage system an address associated with theunit of data.
 52. A method of processing data in a computer systemcomprising at least one host and at least one storage system, the methodcomprising acts of: (a) receiving, at the at least one storage system, afirst request from the at least one host to access a unit of data, therequest having an identifier that can be used to access the unit ofdata; (b) in response to the first request, returning to the at leastone host an identifier that can be used to access the unit of data andadditional information, separate from the identifier, related to wherethe unit of data is physically stored on the at least one storagesystem; and (c) receiving a second request to access the unit of data,the second request including the identifier associated with the unit ofdata and the information related to where the data is physically storedon the storage system.
 53. The method of claim 52, wherein the firstrequest to access the unit of data includes a request to write the unitof data and the act (a) further comprises acts of receiving the unit ofdata from the host and storing the unit of data at a storage location ofthe storage system.
 54. The method of claim 52, further comprising anact of using the information related to where the data is physicallystored on the storage system received in the act (c) to retrieve theunit of data.
 55. The method of claim 52, wherein the at least onestorage system includes a plurality of storage nodes and wherein theinformation related to where the unit of data is physically stored onthe at least one storage system includes information related to on whichof the plurality of storage nodes the unit of data is stored.
 56. Themethod of claim 52, wherein the at least one storage system includes aplurality of storage nodes, at least some of the plurality of storagenodes including a plurality of disks, and wherein the informationrelated to where the unit of data is physically stored on the storagesystem includes information related to on which of the plurality ofdisks the unit of data is stored.
 57. At least one computer readablemedium encoded with instructions that, when executed on a computersystem perform a method of processing data, wherein the computer systemcomprises at least one host and at least one storage system, the methodcomprising acts of: (a) receiving, at the at least one storage system, afirst request from the at least one host to access a unit of data, therequest having an identifier that can be used to access the unit ofdata; and (b) in response to the first request, returning to the atleast one host an identifier that can be used to access the unit of dataand additional information, separate from the identifier, related towhere the unit of data is physically stored on the at least one storagesystem; and (c) receiving a second request to access the unit of data,the second request including the identifier associated with the unit ofdata and the information related to where the data is physically storedon the storage system.
 58. The at least one computer readable medium ofclaim 57, wherein the first request to access the unit of data includesa request to write the unit of data and the act (a) further comprisesacts of receiving the unit of data from the host and storing the unit ofdata at a storage location of the storage system.
 59. The at least onecomputer readable medium of claim 57, wherein the method furthercomprises an act of using the information related to where the data isphysically stored on the storage system received in the act (c) toretrieve the unit of data.
 60. The at least one computer readable mediumof claim 57, wherein the at least one storage system includes aplurality of storage nodes and wherein the information related to wherethe unit of data is physically stored on the at least one storage systemincludes information related to on which of the plurality of storagenodes the unit of data is stored.
 61. The at least one computer readablemedium of claim 57, wherein the at least one storage system includes aplurality of storage nodes, at least some of the plurality of storagenodes including a plurality of disks, and wherein the informationrelated to where the unit of data is physically stored on the storagesystem includes information related to on which of the plurality ofdisks the unit of data is stored.
 62. A storage system for use in acomputer system, including the storage system and at least one host, thestorage system comprising: at least one storage device to store datareceived from the at least one host; and at least one controller that:receives, at the at least one storage system, a first request from theat least one host to access a unit of data, the request having anidentifier that can be used to access the unit of data; and in responseto the first request, returns to the at least one host an identifierthat can be used to access the unit of data and additional information,separate from the identifier, related to where the unit of data isphysically stored on the at least one storage system wherein the atleast one controller receives a second request to access the unit ofdata, the second request including the identifier associated with theunit of data and the information related to where the data is physicallystored on the storage system.
 63. The storage system of claim 62,wherein the first request to access the unit of data includes a requestto write the unit of data, and the at least one controller receives theunit of data from the host and stores the unit of data on the at leastone storage device.
 64. The storage system of claim 62, wherein the atleast one controller uses the information received in the second requestto retrieve the unit of data.
 65. The storage system of claim 62,wherein the at least one storage device comprises a plurality of storagenodes, wherein the information related to where the unit of data isphysically stored on the at least one storage system includesinformation related to on which of the plurality of storage nodes theunit of data is stored.
 66. The storage system of claim 62, wherein theat least one storage device comprises a plurality of storage nodes, atleast some of the plurality of storage nodes including a plurality ofdisks, wherein the information related to where the unit of data isphysically stored on the storage system includes information related toon which of the plurality of disks the unit of data is stored.
 67. Amethod of processing data in a computer system comprising at least onehost and at least one storage system, the method comprising acts of: (a)sending, to the at least one storage system, a first request from the atleast one host to access a unit of data, the first request having anidentifier that can be used to access the unit of data; (b) receiving,from the at least one storage system, an identifier that can be used toaccess the unit of data and additional information, separate from theidentifier, related to where the unit of data is physically stored onthe at least one storage system; and (c) sending a second request toaccess the unit of data from the at least one host to the at least onestorage system, the second request including the identifier associatedwith the unit of data and the information related to where the data isphysically stored on the storage system.
 68. The method of claim 67,wherein the first request to access the unit of data includes a requestto write the unit of data and the act (a) further comprises acts ofsending the unit of data from the at least one host for storage at astorage location of the storage system.
 69. At least one computerreadable medium encoded with instructions that, when executed on acomputer system, perform a method of processing data, wherein thecomputer system comprises at least one host and at least one storagesystem, the method comprising acts of: (a) sending, to the at least onestorage system, a first request from the at least one host to access aunit of data, the first request having an identifier that can be used toaccess the unit of data; (b) receiving, from the at least one storagesystem, an identifier that can be used to access the unit of data andadditional information, separate from the identifier, related where theunit of data is physically stored on the at least one storage system;and (c) sending a second request to access the unit of data from the atleast one host to the at least one storage system, the second requestincluding the identifier associated with the unit of data and theinformation related to where the data is physically stored on thestorage system.
 70. The at least one computer readable medium of claim69, wherein the first request to access the unit of data includes arequest to write the unit of data and the act (a) further comprises actsof sending the unit of data from the at least one host for storage at astorage location of the storage system.
 71. A host computer for use in acomputer system including the host computer and at least one storagesystem, the host computer comprising: at least one storage device; andat least one controller that: sends, to the at least one storage system,a first request from the at least one host to access a unit of data, therequest having an identifier that can be used to access the unit ofdata; and in response to the first request, receives from the at leastone storage system an identifier that can be used to access the unit ofdata and additional information, separate from the identifier, relatedto where the unit of data is physically stored on the at least onestorage system, and stores the additional information on the at leastone storage device; wherein the at least one controller is adapted tosend, along with a second request to access the unit of data from the atleast one host to the at least one storage system, the identifierassociated with the unit of data and the information related to wherethe data is physically stored on the storage system.
 72. The hostcomputer of claim 71, wherein the first request to access the unit ofdata includes a request to write the unit of data and the at least onecontroller sends the unit of data from the at least one host to the atleast one storage system.